Inverse Reinforcement Learning via Ranked and Failed Demonstrations

In many robotics applications, applying reinforcement learning (RL) can be especially difficult, as it depends on the prespecification of a reward function over the environment’s states, which is often hard to define. Inverse Reinforcement Learning (IRL) [1] attempts to address this problem, by utilizing human demonstrations to learn the reward function, without having a human explicitly define it. However, IRL has the underlying assumption that the human demonstrator performs optimally, which can be an unrealistic assumption in extremely complex tasks. Attempts to mitigate this optimal demonstration assumption have been proposed, such as active learning and interactive corrections, but these methods often assume that task execution can be paused, or that the demonstrator can provide the optimal action when queried. With the goal of developing an IRL framework that can perform as successfully as the current state-of-the-art algorithms in reinforcement learning, we introduce a new algorithm, IRLFR, that combines inverse reinforcement learning from failure (IRLF) and inverse reinforcement learning from ranked demonstrations to create a more general framework that can address difficult problems in robot learning with suboptimal demonstrations, so that robots can perform at a superhuman level while minimizing the need for interactive corrections and active learning. We validate our approach in a gridworld domain, where it is nonintuitive for humans to play optimally, and where existing reinforcement learning algorithms can easily find the optimal policy. Our results show that IRLR and IRLFR are frameworks that on average, perform better than the MEIRL baseline performance.